Dynamic reduction in playout of replacement content to help align end of replacement content with end of replaced content

ABSTRACT

When a device is playing out a media stream, the device determines a target time at which the device is to start playing out replacement content in place of a portion of the media stream. However, the device then detects that a starting time when the device starts playout of the replacement content is delayed from the target time by a delay period of duration P. In response, the device then reduces its playout of the replacement content by duration P, to help playout of the replacement content end at a desired time. For instance, the device could seek forward in playout of the replacement content by duration P and/or could remove one or more other portions of the replacement content to total a reduction of duration P.

REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/765,276, filed Aug. 17, 2018, the entirety of which is herebyincorporated by reference.

This application is also related to another U.S. patent applicationfiled by the same applicant on the same date as this application,entitled “Dynamic Playout of Transition Frames While TransitioningBetween Playout of Media Streams,” and having Ser. No. 16/183,185, theentirety of which is hereby incorporated by reference.

BACKGROUND

A typical media client operates to receive and play out (e.g., outputfor playout) an analog or digital media stream representing mediacontent such as video and/or audio content and to output the mediacontent and/or forward the stream for presentation of the content on auser interface such as a display screen and/or an audio speaker.Examples of such clients include televisions, computer monitors,projection systems, loudspeakers, headphones, set top boxes (e.g. cableor satellite TV receivers), digital video recorders, radios, personalcomputers, gaming consoles, streaming media players, and the like.

By way of example, the media client could be a television, which couldreceive a media stream (e.g., an over the air broadcast stream, a mediastream output from a set top box, a media stream played from a digitalstorage medium, an Internet stream, or another such stream) and couldpresent the media content of that stream to a user. As another example,the media client could be a set top box or digital video or audiorecorder, which could receive a broadcast stream from a multi-channelvideo program distributor and/or from data storage and could output themedia content of that stream via a High-Definition Multimedia Interface(HDMI) cable or other interface to a television and/or audio/videoreceiver for playout. And as yet another example the media client couldbe a loudspeaker or headphones, which could receive an audio stream froma radio, computer, or other receiver or audio storage device and couldpresent the audio content of that stream to a user. Numerous otherexamples are possible as well.

SUMMARY

As a media client is playing out a media stream, it may be useful incertain situations for the media client to replace a portion of thestream's media content. For example, it may be useful for the client todynamically replace an ad or other segment of the media content with areplacement ad or with other replacement content, so that as the contentis being presented to a user, the user would receive the replacement adin place of the originally included ad.

Further, it may be desirable for the client to perform such contentreplacement at a specific time point within the media stream. Forinstance, if the client is to dynamically replace an existing ad with areplacement ad, it may be desirable for the client to position thereplacement ad at a time in the media stream when the existing ad wouldhave started.

More particularly, the media stream at issue could define a sequence offrames, such as video frames and/or audio frames, and it may bedesirable for the client to perform the content replacement starting ata particular frame of the media stream. For instance, for adreplacement, it may be desirable for the client to insert thereplacement ad in place of the existing ad, starting at the first frameof the existing ad.

To facilitate such content replacement, the client could learn thespecific time point in the media stream where the client should startthe content replacement, and the client could learn what replacementcontent that the client should start playing at the indicated timepoint. At the desired content-switching time point, the client couldthen transition from playing out the existing media stream to playingout the replacement content as a replacement media stream. Further, whenthe client finishes playing out the replacement media stream, the clientcould then transition back to playout of the original media stream(e.g., in progress).

One technical difficulty with this process is that it may take some timefor the client to transition from playing out a first stream to playingout a second stream, e.g., to start a content replacement, or to switchback to the original stream at the end of a content replacement. Thiscould be the case as the client may need to engage in severalprocessing- and memory-intensive steps before it can start to play outthe second stream. For instance, the client may need to clear caches,receive and decode at least an initial portion of content of the secondstream, and populate a memory buffer with the decoded content forrendering.

In practice, this stream-transition process could take well more thanone frame's worth of time. For instance, if the media at issue is videoat a frame rate of 30 frames per second, each frame would span 1/30 of asecond, i.e., about 33.3 milliseconds (ms). Yet the act of transitioningfrom one video stream to another may take on the order of hundreds ofmilliseconds, thus spanning numerous video frames' worth of time.

Although the client could work to account for this timing by startingits stream-transition process sufficiently far in advance of the desiredcontent-switching time point, the actual amount of time that thetransition process will take may be unpredictable for various reasons.Therefore, it may be unclear precisely how early the client should startthe transition process.

Unfortunately, if the client starts the transition process too late,then the client might not finish the process by the desiredcontent-switching time point, which could give rise to user-experienceissues, as the playout of the second stream may start too late andconsequently may end too late.

For instance, in an ad-replacement scenario where the client is tosubstitute a replacement ad for an original ad of the same duration in alinear broadcast feed, if the client starts playing out the replacementad too late, then the client may end up playing out some of the originalad and then abruptly switching to playout of the replacement ad.Further, if the client plays out the replacement ad for its fullduration, then the replacement ad may continue beyond the end of theoriginal ad, possibly cutting off some content of the underlyingbroadcast feed after the original ad.

To help address this issue, per the present disclosure, the client coulddetect that the time point when the client will start playing thereplacement content is too late, and the client could determine how longthe delay is from the desired start point. The client could thenstrategically reduce its playout of the replacement content in a mannerthat will cause playout of the replacement content to end on time.

For instance, the client could automatically seek forward in thereplacement content by the determined delay duration and start playoutof the replacement content there. Alternatively, the client couldautomatically truncate the end of the replacement content to remove thedetermined delay duration from the end, or the client could selectivelyremove one or more frames (e.g., duplicate frames and/or inter-frames ifapplicable) from within the replacement content. Further, the clientcould carry out a combination of these and/or other such operations. Forinstance, the client could remove some portions of the replacementcontent from the beginning, middle, and end, to total a reduction by thedetermined delay duration.

Further, to help cope with playout of the replacement content startingafter the start of the underlying content that is to be replaced, theclient could also strategically play out some transition frames in placeof that underlying content, until the client is ready to start playingout the replacement content.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description, with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescriptions provided in this summary and below are intended toillustrate the invention by way of example only and not by way oflimitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of system in which disclosedfeatures can be implemented.

FIG. 2 is a diagram illustrating an ad-replacement scenario whereplayout of a replacement ad starts late.

FIG. 3 is a diagram illustrating seeking ahead in the replacement ad ofFIG. 2 to reduce playout time by the delay duration.

FIG. 4 is a diagram illustrating a result of the mechanism illustratedin FIG. 3.

FIG. 5 is a diagram illustrating ending playout of the replacement adearly by the delay duration.

FIG. 6 is a diagram illustrating a result of the mechanism illustratedin FIG. 5.

FIG. 7 is flow chart depicting a method that could be carried out inaccordance with the disclosure.

FIG. 8 is another flow chart depicting a method that could be carriedout in accordance with the disclosure.

FIG. 9 is a simplified block diagram depicting components of an examplemedia client that could operate in accordance with the disclosure.

DETAILED DESCRIPTION

Referring to the drawings, FIG. 1 is a simplified block diagram of anexample system in which various disclosed principles can be applied. Itwill be understood, however, that this and other arrangements andprocesses described herein can take various other forms. For instance,elements and operations can be re-ordered, distributed, replicated,combined, omitted, added, or otherwise modified. Further, it will beunderstood that operations described herein as being carried out by oneor more entities could be implemented by and/or on behalf of thoseentities, through hardware, firmware, and/or software, such as by one ormore processing units executing program instructions or the like.

As shown in FIG. 1, the example system includes a media client 12configured to receive a media stream in real-time from a media source14.

The media client 12 could be configured to operate as described above toreceive from the media source 14 an analog or digital media streamrepresenting media content such as video and/or audio content and tooutput the media content and/or forward the stream for presentation ofthe content on a user interface such as a display screen and/or an audiospeaker. The media client could take any of the forms described above,among other possibilities.

The media source 14 could also take various forms, possibly depending onthe form of the media client 12. By way of example, if the media client12 is a television, the media source 14 could be a local set top box(e.g., cable or satellite receiver) or a streaming media server,configured to output a selected media stream for presentation by thetelevision, or the media source 14 could be a broadcast televisionstation or the like, among other possibilities. And if the media clientis a computer, the media source may comprise an Internet streaming mediaserver or other media provider. Alternatively, the media source 14 couldbe data storage integrated with and/or accessible locally by the mediaclient 12, with the media client receiving (e.g., retrieving) the mediastream from the data storage for playout, such as when the client isplaying out the media stream or is forwarding the media stream forplayout. Other examples are possible as well.

As further shown in FIG. 1, media client 12 could have a communicationinterface with a network (e.g., the Internet or a private network) 16through which the media client can engage in communication with one ormore servers 18. Servers 18 could include file servers, media servers,and/or other types of content servers or the like. Further, althoughmedia source 14 is shown separately from servers 18, media source 14could alternatively be included or integrated with one or more ofservers 18 and could be accessible via network 16.

As noted above, when a media client such as client 12 is receiving andplaying out (e.g., outputting for playout) a media stream, it may beuseful for the client to replace a portion of the stream withreplacement content defining a replacement stream. Further, as notedabove, it may be desirable for the client to do this at a specific timepoint within the media stream. For instance, if the stream defines asequence of frames, such as video frames and/or audio frames, it may bedesirable for the client to perform the content replacement starting ata specific frame of the sequence.

In practice, the client could learn (determine) in various ways thespecific target time point in the media stream where the client shouldstart the content replacement.

For instance, the client could be provisioned with, or receive along orin association with the media stream, a set of data that identifies ortimestamps each frame within the media stream (e.g., timestamps embeddedin the media stream or otherwise provided through in-stream signaling)as well as data that specifies the identity or timestamp of the framewhere the content replacement is to start. Based on that data, theclient could thus determine the specific time point in the media streamwhere the content replacement is to start.

Alternatively, as the client receives and plays out the media stream,the client could work with a network server to learn the point in themedia stream where the content replacement is to start, as a desiredcontent-switching time point. For instance, the server could firstidentify the media stream being played out by the client. And uponidentifying the media stream, the server could then determine a specifictime point in the media stream where the client should start the contentreplacement and could inform the client of that determined time point.

To facilitate the server's identification of the media stream beingplayed out by the client, the client (or perhaps an adjunct device)could generate digital fingerprint data representing the media contentin real time as the client receives the stream, and the server couldcompare that fingerprint data with reference fingerprint datarepresenting media content of a known media stream, with a fingerprintmatch indicating that the media stream being played by the client is theknown media stream.

The server could establish or otherwise have access to referencefingerprint data representing the known media stream. For instance, theserver or an associated watching station or other entity could receive abroadcast of the known media stream and, while receiving the broadcast,could generate a sequence of reference digital fingerprints representingframes of the known media stream. The server or another entity couldestablish these reference fingerprints by programmatically analyzingframes of the known media stream, receiving as input a copy of the knownmedia stream and applying any media fingerprinting process or algorithmnow known or later developed to generate the reference fingerprints. Orthe server could otherwise establish or be provisioned with thissequence of reference fingerprints representing the known media stream.The server could then store the reference fingerprints in associationwith an identity of the known media stream.

Further, in real time as the client plays out the media stream, theclient could generate query fingerprints representing frames of thatstream and could transmit the query fingerprints to the server foranalysis. To facilitate fingerprint comparisons, the client couldgenerate these query fingerprints using the same fingerprinting processor algorithm that is or was applied to establish the referencefingerprints. In addition, the client could generate the queryfingerprints at a particular frame rate and could transmit thefingerprints in groups to the server. For instance, for a media streamcomprising video at 30 frames per second, the client might generate thequery fingerprints at a reduced frame rate on the order of 2 frames persecond to help conserve resources, and the client might transmit alatest group of query fingerprints to the server every 1.5 seconds orso. Other examples are possible as well.

As the server receives the query fingerprint data from the client, theserver could compare the query fingerprint data with the referencefingerprint data of the known media stream. And if the server therebydetermines with sufficient confidence that the query fingerprint datamatches the reference fingerprint data, the server could conclude thatthe media stream being played out by the client is the known mediastream.

The server could conduct this fingerprint comparison on a per-framebasis. Considering video content, for instance, the server could conductthe analysis on a per-video-frame basis, comparing a query fingerprintof a frame against a reference fingerprint of a frame. The comparisonprocess could take any of a variety of forms, depending on the structureof the fingerprints. For example, the comparison could involve computinga maximum deviation between the fingerprints and determining if themaximum deviation is within a predefined tolerance. Further, if thefingerprints are binary, this could be a Boolean determination or couldinvolve computing a Hamming distance (as a count of mismatches betweenrespective bit locations in the fingerprints). And if the fingerprintsare more complex values, such as decimal values or vectors (e.g., greyvalues per video frame region), this could involve determining adistance between the values or vectors. Numerous other examples arepossible as well.

In addition, the server could conduct this fingerprint comparisonprocess with a desired level of tolerance. For instance, the comparisoncould be coarse, with relatively high tolerance for differences betweenthe fingerprints, such as with tolerance for a greater number ofdifferences and/or tolerance for a greater size of differences. By wayof example, when comparing two 32-bit fingerprints with each other, acoarse comparison may deem the fingerprints to match each othergenerally even if two or another designated number of bits do not match.Or the comparison could be more granular, with lower tolerance for suchdifferences, and/or with comparisons focused on specific fingerprintcomponents possibly representing specific portions of the underlyingmedia.

Through this comparison of the client-generated query fingerprints withthe reference fingerprints of the known media stream, the server couldthereby find one or more matches. For instance, the server could findone or more instances where a query fingerprint of a frame of the mediastream being played out by the client matches a respective referencefingerprint of a frame of the known media stream. This matching couldthus establish or support a conclusion that the media stream beingplayed out by the client is the known media stream.

Alternatively, the server could identify the media stream at issue inother ways. For instance, if the media stream contains watermark datacorresponding with an identity of the media stream, the client couldread or otherwise extract the watermark data and report the watermarkdata to the server. And the server could then use the reported watermarkdata as a basis to identify the media stream being received by theclient. For example, the server could consult reference data thatcorrelates various watermark data with particular known media streams,to determine that the watermark data reported by the client correspondswith a particular known media stream.

Having identified the media stream being played out by the client, theserver could then determine a time point in the media stream where acontent replacement is to occur. For instance, the server could consultreference data that specifies a time point within the known media streamwhere a content replacement is to start. And the server could thentransmit to the client a directive that indicates that time point.

The replacement time point specified by the server's reference datacould be defined in a manner that is understandable to the client. Forinstance, if the client is provisioned with data that timestamps eachframe of the media stream along a “true time” timeline defined withinthe media stream, the replacement time point that the server has mightbe the true-time timestamp of an upcoming frame of the media stream. Theserver could thus provide the client with that true-time timestamp as anindication of the time when the client should start the contentreplacement.

Further, the client could map that true-time timestamp to “client time”defined according to the client's internal clock so that the client candetermine a time on its own clock when it should start the contentreplacement. For instance, the client could (i) compute a differencebetween the true-time timestamp of a current frame and the current timeon the client's clock and (ii) apply that determined difference to theindicated true-time timestamp of the upcoming frame where thereplacement is to occur, to determine the client-time point when theclient should start the content replacement.

Alternatively, the replacement time point specified by the server'sreference data might be defined in a manner that is not understandableto the client. For instance, the replacement time point that the serverhas might be a timestamp defined according to a true-time timelinewithin the known media stream, but the client might have no context orperspective for true time. In that case, the server could engage in aprocess to establishing a mapping or “synchronous lock” between truetime and client time, and the server then map the true-time time pointwhere the content replacement is to occur to a corresponding client-timetime point and provide that client-time time point to the client.

As an non-limiting example of this, consider a media stream that definesan ad pod that could appear within a television broadcast. An ad podcomprises a sequence of ads, such as television commercials. With thisexample, the client might be a television that is receiving and playingout a linear broadcast feed of a particular television channel, and thead pod might appear at any moment within that feed. Especially with alive broadcast such as a sporting event, for instance, the time when thead pod appears in such a feed may be unpredictable. Alternatively, thead pod could appear at a specifically scheduled time.

In this example, the desired content replacement could be replacement ofa particular ad within the ad pod (e.g. substituting for the ad adifferent ad or other replacement content). For instance, if the ad podis a sequence of ads A1, A2, A3, A4, the desired content replacementcould be replacement of ad A3.

In this situation, the ad pod may define a true-time timeline startingat time T=0 and continuing to the end of the ad pod, and ad A3 may beginat time T_(A3) on that timeline. However, the client may lack contextfor that timeline. The client may have its own internal clock that has atimeline (e.g., time of day or other internal time reference) definingclient time. But without advanced knowledge of where the ad-pod timelineexists within client time, merely given the value T_(A3) as the desiredcontent-revision time point would not enable the client to start the adreplacement when desired.

For the server to establish synchronous lock between true time andclient time in that context, the client could timestamp the queryfingerprints that the client sends to the server. In particular, as theclient plays out the media stream and generates query fingerprints, theclient could timestamp each query fingerprint with a current client-timevalue. When the server detects one or more matches between the queryfingerprints and timestamped reference fingerprints of the known mediastream, the server could then use those matching points as a basis tomap true time to client time. For instance, based on at least one suchdetected matching point, the server could define a mapping between (i)the true-time timestamp of the matching reference fingerprint and (ii)the client-time time stamp of the matching query fingerprint.

Once the server has established synchronous lock, the server could thenuse the established synchronous lock to translate the true-time timepoint where the content replacement is to occur into a correspondingclient-time time point at which the content replacement is to occur. Andthe server could transmit that determined client-time time point to theclient, to enable the client to start the content replacement at thedesired time point in the media stream that the client is playing out.

Further, in an alternative implementation, the client rather than theserver could carry out the process of establishing and applyingsynchronous lock in order to determine the client time point where thecontent replacement is to occur.

For instance, once the server has identified the media stream beingreceived by the client, the server could transmit to the client somereference data defining reference fingerprints and true-time timestampsfor an upcoming portion of the known media stream. The client could thenconduct the fingerprint matching between query fingerprints andreference fingerprints to find one or more matching points, and theclient could establish synchronous lock between true time and clienttime by similarly correlating the timestamps of at least one matchingpoint. Once the client has established the synchronous lock, the clientcould then use the synchronous lock as a basis to translate a providedtrue-time time point where the content replacement is to occur into acorresponding client-time time point where the content replacement is tooccur.

In addition to determining the target time point in the media streamwhen the client should start the content replacement, the client couldalso learn in various ways what content replacement the client shouldperform starting at that time point.

By way of example, the client could be pre-provisioned in advance with aspecification of the content replacement to start at the indicated timepoint, such as an indication that the client should substitute certaincontent in place of existing content of the media stream. Alternatively,the client could receive from a server or another entity a specificationof the content replacement to perform. For instance, when a servertransmits to the client a directive informing the client of the timepoint in the media stream when the client is to start the contentrevision, the server could include in that directive (or provideseparately) to the client a specification of the content replacement toperform.

In practice, the specification of the content replacement to performcould comprise a pointer to replacement content that the client is toobtain and is to insert in place of or as a supplement to existingcontent of the media stream. For example, as noted above, thespecification could comprise a Universal Resource Locator (URL) thatdefines or correlates with a network address of a streaming mediaserver, file server, or the like, from which the client could requestand receive the replacement content. Alternatively, the specificationcould indicate in another way where and/or how the client is to obtainthe replacement content, to facilitate performing the contentreplacement starting at the indicated time point in the media stream.Further, the specification could indicate a duration of the replacementcontent, to inform the client how long the client should play thereplacement content in place of the original content.

At the indicated time point where the content replacement is to start,the client could thus switch from playing out the existing media streamto playing out the replacement content as a replacement media stream.Further, at the end of the content replacement, the client could thenswitch back to playing out the original media stream (e.g., inprogress).

As a non-limiting example, consider again the ad pod discussed above. Inpractice, a server could be provisioned with timestamped referencefingerprints representing and corresponding with the frames of the adpod. Further, the server could be provisioned with an indication that aparticular one of the ads in the ad pod should be replaced, and withinformation to facilitate that ad replacement. For instance, the servercould be provisioned with an indication of the true-time point of thefirst frame of the existing ad, a URL of a replacement ad, and anindication of the duration of the replacement.

On an ongoing basis as the client receives and plays out a linearbroadcast feed that may contain this ad pod, or in response to one ormore other triggers, the client could be generating timestamped queryfingerprints of the broadcast feed and could be reporting thosetimestamped query fingerprints to the server for analysis. Comparingsuch query fingerprints with the reference fingerprints representing theframes of the ad pod sequence, the server could thereby detect that theclient is playing out the ad pod sequence.

Having determined that the client is playing out the ad pod sequence,the server could then translate the true-time point of the frame wherethe ad replacement is to start into a corresponding client-time point inthe manner discussed above for instance and could transmit to the clienta directive that specifies that client-time point, includes a pointer toa replacement ad that the client is to substitute for frames of themedia stream starting at that client-time, and directs the client toperform the indicated content replacement. Accordingly, the client couldobtain the indicated replacement ad and could substitute that ad forframes of the media stream starting at the indicated client-time point.

Further, the server could inform the client of a duration of the adreplacement or of a client-time point when the ad replacement is to end,or that duration could be a standard or pre-set duration. And at the endof the ad replacement, the client could then switch back from playingout the replacement ad to playing out the broadcast feed (e.g., aremainder of the ad-pod sequence) in progress.

As explained above, a technical problem that can arise in this processis that it may take some time for the client to transition from playingout one stream to playing out another stream, such as from playing outan ad-pod sequence in a linear broadcast stream to playing out areplacement ad stream. In particular, as noted above, the client mayneed to engage in various processes, such as clearing caches, receivingand decoding content, and populating memory buffers, to facilitatestarting to play out the replacement stream.

The starting point of this stream-transition process could be defined invarious ways. For instance, if the client will receive content of thesecond stream from a streaming media server, the start of thestream-transition process could be when the client sends an initialTransport Control Protocol (TCP) handshake message to initiate setup ofa TCP session with the server or when the client sends a Real-TimeStreaming Protocol (RTSP) SETUP request or the like to initiate setup ofa real-time transport protocol (RTP) session with the server, amongother possibilities. Alternatively, if the client would receive contentof the second stream in another manner or from another source, thestarting point of the process could be defined in other ways. The end ofthe process could then be defined as the point in time when the clienthas buffered enough of the second media stream to start playing thestream while accounting for jitter and/or other issues. Other examplesare possible as well.

As noted above, the stream-transition process, from start to finish, maytake on the order of hundreds of milliseconds. Further, as resourceconstraints and other factors change from time to time, the actualamount of time that it will take the client to transition from onestream to another in any given instance may be unpredictable.

Experience may establish minimum and maximum expected stream-transitiondurations for the client (or for a representative client). For instance,testing might show that it will take anywhere from 300 ms to 500 ms forthe client to transition from playout of a first stream to playout of asecond stream (e.g., to the point where the client is ready to startplaying the second stream). Thus, the client may have a minimum expectedswitching duration of 300 ms and a maximum expected transition durationof 500 ms, defining a 200 ms uncertainty range, among otherpossibilities.

The client could determine these minimum and maximum expectedstream-transition durations by tracking stream-transition durations overtime and rolling up the tracked durations to establish statisticalmeasures of minimum and maximum, and/or the client could otherwise beprovisioned with representative values of the expected transitiondurations.

Given this uncertainty range, the client could configure its switchingprocess to start in advance of the desired content-switching time pointby an amount of time equal to the maximum expected switching duration.That way, the client should theoretically end up starting to play outthe second stream sometime within the uncertainty range, at or beforethe desired content-switching start time.

Unfortunately, however, once the client has finished its preparationsfor playing out the second stream, the client may then no longer be in aposition to play out the first stream, since the client may now bebuffering the second-stream content for playout rather than bufferingthe first-stream content for playout. Therefore, if the client finishesits preparations for playout out the second stream before the end of themaximum expected switching duration has passed, then the client maystart playout of the second stream earlier than desired, which mayresult in user-experience issues. Further, for content-replacement suchas ad replacement, this premature starting of the playout of the of thesecond stream might create associated user-experience issues at the endof the replacement duration when the client is to switch back fromplayout of the second stream to playout of the first stream.

One way to help avoid having the client start playing out the secondstream earlier than desired is for the client to instead configure itsswitching process to start in advance of the desired content-switchingtime point by an amount of time that is less than the maximum expectedswitching duration. For instance, the client could start its switchingprocess in advance of the desired content-switching time point by anamount of time that is less than the maximum expected switching durationbut greater than or equal to the minimum expected switching duration.

By way of example, the client could start its switching process inadvance of the desired content-switching time point by an amount of timethat is an average expected switching duration, such as half way betweenthe minimum expected switching duration and the maximum expectedswitching duration. Using the example values above, where the minimumexpected switching duration is 300 ms and the maximum expected switchingduration is 500 ms, for instance, the client could thus start itsswitching process 400 ms before the desired content-switching timepoint.

Yet this solution is also not optimal, since it may ultimately take theclient longer than that average expected switching duration to preparefor playout of the replacement content. Consequently, the client mightnot finish its content-switching preparations until after the desiredcontent-switching time point has passed. And as a result, as notedabove, the client may end up playing some of the underlying content thatis supposed to be replaced and then abruptly switching to playout of thereplacement content, which may provide a poor user experience. Further,as noted above, if the client starts playing out the replacement contenttoo late and plays the replacement content for its full duration, thenplayout of the replacement content may last beyond the end of thecontent that was supposed to be replaced and may cut off the start ofsucceeding content of the underlying media stream.

The present disclosure provides a mechanism to help address thetechnical problem of playout of replacement content starting too lateand therefore ending too late.

Per the disclosure, when the client has started or will start playingout replacement content with some delay after a desiredcontent-switching time point has passed, the client will determine howlong the delay is. For example, the client could determine how long haspassed or will have passed from the desired content-switching time pointto the time when playout of the replacement content started or willstart. And as noted above, the client will then strategically reduce itsplayout of the replacement content in a manner that will cause itsplayout of the replacement content to end on time. Further, to help copewith playout of the replacement content starting after the start of theunderlying content that is to be replaced, as noted above, the clientcould also strategically play out some transition frames in place ofthat underlying content, until the client is ready to start playing outthe replacement content.

This process assumes that the content to be replaced (the originalcontent) and the replacement content are the same duration D as eachother, and that—absent any delay—the goal would be to start playout ofthe replacement content precisely in time when the original contentwould have started and to end playout of the replacement contentprecisely in time when the original content would have ended. Forinstance, in an ad-replacement scenario, an original ad and areplacement ad may each be 30 seconds or another typical ad duration,and the goal may be to start playout of the replacement ad preciselywhen playout of the original ad would have started and to end playout ofthe replacement ad precisely when playout of the original ad would haveended.

Further, the process assumes that the client's start to playout of thereplacement content is, was, or will be delayed by a delay period ofduration P seconds (perhaps several frames spanning just tens ofmilliseconds, possibly less than a second) after the time when theoriginal content would have started, i.e., after the desiredcontent-switching time point. The client could detect the existence ofsuch a delay by detecting that the client has not started, or will notstart, playout of the replacement content by the desiredcontent-switching time point. And the client could compute the length ofthe delay by programmatically running a stopwatch from the when thedesired content-switching time point arrives until the time when theclient starts playing out the replacement content.

As the desire is for the client to then finish its playout of thereplacement content precisely when the original content would haveended, the client will then be left with a reduced period of D-P (i.e.,the difference between the full ad duration D and the delay period P) inwhich to play out the replacement content.

Given that the replacement content is of duration D, the client couldfacilitate playout of the replacement content in reduced period of D-Pby removing P seconds from the replacement content, i.e., by forgoingplayout of P seconds of the replacement content. In accordance with thepresent disclosure, the client could do this by removing P seconds ofthe start of the replacement content, removing P seconds of the end ofthe replacement content, or removing one or more portions of thereplacement content totaling P seconds.

As one example of this process, upon determining that playout of thereplacement content is delayed by P seconds, the client couldresponsively seek forward within the replacement content (i.e., jumpahead in the replacement content stream) by P seconds and start playoutof the replacement content at the P-second time point. That way, theclient's playout of the remainder of the replacement content should takejust D-P seconds, so that the playout will end on time. With thisexample, since the client will be starting playout of the replacementcontent at a point that is later than the actual start of thereplacement content, the client could also add a fade-in (e.g.,dissolve) to playout of the replacement content so that the starting ofplayout of the replacement content does not seem so abrupt.

The client could carry out this seeking operation starting precisely atthe beginning of the replacement content. Alternatively, the clientmight start its seeking-forward at a point after (perhaps slightlyafter) the beginning of the replacement content and could still jumpahead by P seconds in playout of the replacement content.

As another example of this process, upon determining that playout of thereplacement content is delayed by P seconds, the client couldresponsively end playout of the replacement content P seconds early.That is, the client could start playout of the replacement content atthe start of the replacement content but, rather than playing thereplacement content for the fully duration of D seconds, could stopplayout of the replacement content (and switch back to the underlyingmedia stream) once the client has played out D-P seconds of thereplacement content. As playout of the replacement content will thustake just D-P seconds, the playout should likewise end on time. Further,with this example, since the client will be ending playout of thereplacement content early, the client could also add a fade-out from thereplacement content back to the underlying media stream so that theending of playout of the replacement content will not seem so abrupt.

As yet another example of this process, upon determining that playout ofthe replacement content is delayed by P seconds, the client couldresponsively engage in a process to selectively remove frames fromwithin the replacement content, in an effort to reduce the totalduration of playout of the replacement content to D-P seconds.

For instance, as the client plays out the replacement content, theclient could examine upcoming frames of the replacement content that ithas buffered for playout, in search of frames that the client couldreasonably remove and thus not play out.

By way of example, if the media at issue comprises video, the clientcould search for instances where the images of two or more consecutiveframes are identical or sufficiently similar, optimally when there is noaccompanying audio. The client could conduct this analysis by generatingand comparing fingerprints of consecutive frames, by pattern matching,and/or in some other manner. Each time the client finds such repeatframes, the client could then discard one or more duplicate frames.Alternatively or additionally, the client could look for inter-frames(e.g., P-frames or B-frames) of video, also optimally when there is noaccompanying audio, and could similarly remove those frames. Further, ifthe media at issue comprises audio, the client could search for silentframes, possibly repeated silent frames, and could likewise discardthose frames.

Alternatively, the client could apply one or more other processes toselectively remove frames from the replacement content, to help reducethe duration of replacement content by the delay period P.

By selectively carrying out this frame-removal process at possiblymultiple different times within the replacement content, the clientmight be able to more subtly remove a total of P seconds of frames, sothat the resulting duration of playout of the replacement content wouldbe just D-P seconds and would therefore end on time. Alternatively, ifthe client has not been able to remove a sufficient number of frames ofthe replacement content when the client approaches D-P seconds ofplayout of the replacement content, then the client could end playout ofthe replacement content early enough to bring the total duration ofplayout of the replacement content down to D-P seconds, so that it willlikewise end on time.

As further noted above, when the client is going to start playing outthe replacement content with a delay of P seconds, the client could alsoplay out some transition frames in place of the original content duringthat delay period. To facilitate this, when the client detects that thedesired content-switching time point has arrived, the client coulddetermine that it is not yet ready to start playing the replacementcontent. In response, the client could then start generating and playingout transition frames in place of the original content until the clientis ready to start playing out the replacement content. And when theclient is ready to start playing out the replacement content, the clientcould then switch from playing out the transition frames to playing outthe replacement content. Further, the client could add fading (e.g.,dissolving) from playout of the transition frames to playout of thereplacement content, to help smooth that transition.

In practice, the transition frames could take various forms, based onthe type of media at issue. For instance, if the media comprises video(possibly along with audio), then the transition frames could compriseblack frames and/or copies of a frame immediately preceding thetransition. By way of example, the client could play out black framesfor the duration of transition period (until transitioned to play outthe replacement content). Alternatively, for some time into thetransition period, the client could play out frames that are a copy of alatest frame of the first stream and could then fade from that image toblack frames for a remainder of the transition period. Whereas if themedia comprises audio (possibly along with video), then the transitionframes could be silent frames. In particular, the client could presentsilent frames for the duration of the transition period. Other examplesare possible as well.

FIGS. 2-6 illustrate examples of some of the processes discussed above,particularly in an ad-replacement scenario. Each of these figuresdepicts content playout along a timeline, with time progressing fromleft to right.

FIG. 2 depicts the problem scenario where a client is playing out an adpod within a linear stream 20 and where the client is to substituteplayout of a replacement ad 22 for an ad, AD2, of the linear stream, butwhere playout of the replacement ad 22 starts late. As shown, AD2 is ofduration D, and the replacement ad 22 is also of duration D. Further, asshown along the timeline, the ultimate goal would have been to startplayout of the replacement ad precisely at the start of AD2, at timeT_(DESIRED-START), and to end playout of the replacement ad precisely atthe end of AD2, at time T_(DESIRED-END). Yet as shown, the client delaysstarting its playout of the replacement ad by P seconds, as the clientis ready to start playout of the replacement content at time T_(READY),which is P seconds after time T_(DESIRED-START). As a result, in thisscenario, the client may play out P seconds of AD2 and then abruptlyswitch to playing out the replacement ad. Further, the client's playoutof the replacement ad may extend P seconds past the start of the nextad, AD3, thus possibly cutting off the start of that next ad.

FIG. 3 depicts one mechanism for the client to help overcome thesetechnical issues. In particular, FIG. 3 illustrates that the clientcould seek ahead in the replacement ad 22 by P seconds and start playoutof the replacement ad at that point. To facilitate this in practice, theclient could apply a native seek function to jump ahead in thereplacement ad stream or could otherwise discard the first P seconds offrames of the replacement ad. As a result, the client would start itsplayout of the replacement ad at P seconds into the replacement ad, sothat the client would play out just the last D-P seconds of thereplacement ad.

FIG. 4 next illustrates an example result of this seeking process. Asshown in FIG. 4, the client's playout of the last D-P seconds of thereplacement ad will start at time T_(READY) and will end at timeT_(DESIRED-END), so that the end of the client's playout of thereplacement ad should optimally align with the end of AD2 in theunderlying linear stream 20 as desired. (Though not shown, the clientmay also need to take action at or near the end of this replacement adplayout, to account for delay in switching back from playout of thereplacement ad stream 22 to playout of the underlying linear stream 20.For instance, the client could handle that situation by playing outtransition frames and perhaps by seeking forward in playout of AD3 by anappropriate duration.)

As further shown in FIG. 4, in line with the discussion above, theclient could also add transition frames 24 with fade periods to helpaccount for the delayed start of playing out the replacement ad. In anexample implementation, the client could generate and play out thesetransition frames for at least the delay period (from T_(DESIRED-START)to T_(READY)), to avoid playing out the start of AD2 that is supposed tobe replaced by the replacement ad. Further, the client could extendplayout of the transition frames for a fade period 26 after the delayperiod and, through the fade period 26, fade from playout of thetransition frames to playout of the replacement ad, to help smooth thattransition.

FIG. 5 illustrates an alternative mechanism for the client to helpovercome the issues associated with delayed start to playout of thereplacement ad. In particular, FIG. 5 illustrates that the client couldend playout of the replacement ad P seconds early to avoid havingplayout of the replacement ad extend beyond time T_(DESIRED-END). Tofacilitate this in practice, the client could play out just the firstD-P seconds of the replacement ad, and the client could discard the lastP seconds of the replacement ad. For instance, once the client startsplaying out the replacement ad with the delay of P seconds, the clientcould then continue to play out the replacement ad for D-P seconds.

FIG. 6 next illustrates an example result of this alternative process.As shown in FIG. 6, the client's playout of the first D-P seconds of thereplacement ad will start at time T_(READY) and will end at timeT_(DESIRED-END), so that the end of the client's playout of thereplacement ad should optimally align with the end of AD2 in theunderlying linear stream 20 as desired. (Here again, the client maysimilarly need to take action such as fading at or near the end of thisreplacement ad playout, to account for delay in switching back fromplayout of the replacement ad stream to playout of the linear stream.)

FIG. 7 is next a flow chart depicting a method that can be carried outin accordance with this disclosure. This method could be carried out bya device such as a media client or other playback device, to facilitateplayout of replacement content in place of a portion of an underlyingmedia stream, where the replacement content and the portion of theunderlying media stream are of equal duration D. As discussed above, inan example implementation, the portion of the underlying media streamcould be an ad in an ad pod, and the replacement content could be areplacement ad. Alternatively, the content could take other forms.

As shown in FIG. 7, at block 70, while the device is playing out theunderlying media stream, the device determines a target time at which tobegin playout of the replacement content. For instance, the client coulddetermine this target time by receiving a directive from a server orother entity specifying the target time and/or by itself evaluating todetermine the target time.

At block 72, device then detects that the a starting time at which thedevice starts playout of the replacement content is after the determinedtarget time by a delay period of duration P. For instance, the devicecould measure how much time has passed from the desired start time tothe time when the device will start, is starting, or will start playoutof the replacement content.

And at block 74, in response to detecting this delay, the device willthen reduce its playout of the replacement content by duration P to helpalign an end of playout of the replacement content with an end of theportion of the underlying media stream. As discussed above, thisreducing operation could involve seeking forward in the replacementcontent by duration P to a new time point in the replacement content,and starting playout of the replacement content at the new time point.Alternatively or additionally, the reducing could involve ending playoutof the replacement content a duration of P early. Still alternatively oradditionally, the reducing could involve removing one or moreintervening frames from the replacement content, to help bring the totalduration of playout of replacement content down to D-P.

As further discussed above, the device could additionally play outtransition frames during the delay period P, in place of playout of aninitial part of the portion of the underlying media stream. And thedevice could fade from playout of the transition frames to playout ofthe replacement content.

In addition, as discussed above, if the media at issue comprises video,then the transition frames could comprise black frames and/or framesthat are a copy of a last played frame of the first media stream. And ifthe media at issue comprises audio, then the transition frames couldcomprise silent frames.

FIG. 8 is another flow chart depicting a method that can be carried outin accordance with this disclosure. As shown in FIG. 8, at block 80,when a device is playing out a media stream, the device determines atarget time at which the device is to begin playout of replacementcontent in place of a portion of the media stream. At block 82, thedevice detects that a starting time at which the device starts playoutof the replacement content is after the determined target time by adelay period of duration P. And at block 84, responsive to thedetecting, the device seeks forward in the replacement content byduration P to reduce playout of the replacement content by duration P.

Various features described above can be applied in the context of thismethod and vice versa. For instance, the media stream could be an adpod, and the replacement content could be a replacement ad. Further, thedevice could additionally play out transition frames (e.g., blackframes, silent frames, or the like) during the delay period P and couldadd fading to smooth transitions.

Finally, FIG. 9 is a simplified block diagram of an example mediaclient, showing some of the components that can be included in such adevice to facilitate carrying out operations such as those noted above.In line with the discussion above, this client could take various forms.For instance, it could be a television, computer monitor, or otherdevice that operates to receive and render video content, and/or itcould be a loudspeaker, a pair of headphones, or other device thatoperates to receive and render audio content. Numerous other examplesare possible as well.

As shown in FIG. 9, the example client includes a media input interface90, a network communication interface 92, a media presentation interface94 a processing unit 96, and non-transitory data storage 98, any or allof which could be integrated together or, as shown, communicativelylinked together by a system bus, network, or other connection mechanism100.

The client could use media input interface 90 and/or networkcommunication interface 92 to receive media streams for presentation.Either of these interfaces could include one or more wired and/orwireless interfaces for establishing communication with and receivingmedia content in analog or digital form from a receiver, server, orother device or system. For example, the media input interface couldinclude one or more interfaces compliant with protocols such as DVI,HDMI, VGA, USB, BLUETOOTH, WIFI, among numerous others, and the networkcommunication interface could comprise a wired or wireless Ethernetinterface, for engaging in local or wide area packet-based communicationso as to interact with one or more servers and/or other devices orsystems.

Media presentation interface 94 could then comprise one or morecomponents to facilitate presentation of media content. By way ofexample, the media presentation interface could comprise a userinterface such as a display screen and/or a loudspeaker, as well as oneor more drivers or other components for processing the received mediacontent to facilitate presentation of the content on the user interface.

Processing unit 96 could comprise one or more general purpose processors(e.g., microprocessors) and/or one or more specialized processors (e.g.,application specific integrated circuits). And non-transitory datastorage 98 could comprise one or more volatile and/or non-volatilestorage components, such as optical, magnetic, or flash storage.

Further, as shown, data storage 98 could store program instructions 102,which could be executable by processing unit 96 to cause the client tocarry out various operations described herein. For instance, theoperations could include, while the client is playing out an underlyingmedia stream, determining a target time at which the client is to beginplayout of replacement content in place of a portion of the underlyingmedia stream. Further, the operations could include detecting that astarting time at which the client starts (e.g., has started, isstarting, or will start) playout of the replacement content is after thedetermined target time by a delay period of duration P. And theoperations could include, responsive to the detecting, reducing playoutof the replacement content by duration P. Here too, various operationsdescribed above could be applied in this context and vice versa.

Exemplary embodiments have been described above. Those skilled in theart will understand, however, that changes and modifications may be madeto these embodiments without departing from the true scope and spirit ofthe invention.

What is claimed is:
 1. A method for a device to play out replacementcontent in place of a portion of an underlying media stream, wherein thereplacement content and the portion of the underlying media stream areof equal duration D, the method comprising: while the device is playingout the underlying media stream, determining by the device a target timeat which to begin playout of the replacement content; detecting by thedevice that a starting time at which the device starts playout of thereplacement content is or will be after the determined target time by adelay period of duration P; and responsive to the detecting, reducing bythe device playout of the replacement content to help align an end ofplayout of the replacement content with an end of the portion of theunderlying media stream, wherein reducing playout of the replacementcontent includes (i) detecting in the replacement content one or moreduplicate frames, (ii) responsive to detecting in the replacementcontent the one or more duplicate frames, skipping playout of the one ormore duplicate frames when playing out the replacement content, (iii)detecting by the device that skipping playout of the one or moreduplicate frames shortens the playout of the replacement content by lessthan the duration P, and (iv) responsive to the detecting that theremoval of the duplicate frames shortens the playout of the replacementcontent by less than the delay period of duration P, truncating by thedevice playout of the replacement content such that the playout of thereplacement content lasts a duration D-P to help align the end ofplayout of the replacement content with the end of the portion of theunderlying media stream.
 2. The method of claim 1, wherein the portionof the underlying media stream is an ad in an ad pod, and wherein thereplacement content is a replacement ad.
 3. The method of claim 1,further comprising playing out by the device transition frames duringthe delay period, in place of playout of an initial part of the portionof the underlying media stream.
 4. The method of claim 3, furthercomprising fading by the device from playout of the transition frames toplayout of the replacement content.
 5. The method of claim 3, whereinthe underlying media stream comprises video, and wherein the transitionframes comprise black frames.
 6. The method of claim 3, wherein theunderlying media stream comprises audio, and wherein the transitionframes comprise silent frames.
 7. The method of claim 1, wherein thedevice is a television.
 8. The method of claim 1, wherein the device isa set top box.
 9. The method of claim 1, wherein the media streamcomprises watermark data corresponding with an identity of the mediastream, wherein the identity of the media stream is associated with thestart time, wherein the method further comprises: determining, by thedevice from the media stream, the watermark data; and based on thewatermark data, determining by the device the identity of the mediastream, wherein detecting that a starting time at which the media clientstarts playout of the replacement content is or will be after thedetermined target time by a delay period of duration P is based on theidentity of the media stream.
 10. The method of claim 1, furthercomprising: buffering by the device for playout a plurality of upcomingframes of the replacement content, wherein detecting in the replacementcontent one or more duplicate frames comprises detecting the one or moreduplicate frames in the plurality of buffered upcoming frames of thereplacement content.
 11. A media client comprising: a processing unit;non-transitory data storage; and program instructions stored in thenon-transitory data storage and executable by the processing unit tocause the media client to carry out operations including: while themedia client is playing out an underlying media stream, determining atarget time at which the media client is to begin playout of replacementcontent in place of a portion of the underlying media stream, detectingthat a starting time at which the media client starts playout of thereplacement content is or will be after the determined target time by adelay period of duration P, and responsive to the detecting, reducingplayout of the replacement content to help align an end of playout ofthe replacement content with an end of the portion of the underlyingmedia stream, wherein reducing playout of the replacement contentincludes (i) detecting in the replacement content one or more duplicateframes, (ii) responsive to detecting in the replacement content the oneor more duplicate frames, skipping playout of the one or more duplicateframes when playing out the replacement content, (iii) detecting by thedevice that skipping playout of the one or more duplicate framesshortens the playout of the replacement content by less than theduration P, and (iv) responsive to the detecting that the removal of theduplicate frames shortens the playout of the replacement content by lessthan the delay period of duration P, truncating by the device playout ofthe replacement content such that the playout of the replacement contentlasts a duration D-P to help align the end of playout of the replacementcontent with the end of the portion of the underlying media stream. 12.The media client of claim 11, wherein the portion of the underlyingmedia stream is an ad in an ad pod, and wherein the replacement contentis a replacement ad.
 13. The media client of claim 11, wherein theoperations further include playing out transition frames during thedelay period, in place of playout of an initial part of the portion ofthe underlying media stream.
 14. The media client of claim 13, whereinthe operations further include fading by the device from playout of thetransition frames to playout of the replacement content.
 15. The mediaclient of claim 13, wherein the underlying media stream comprises video,and wherein the transition frames comprise black frames.
 16. The mediaclient of claim 13, wherein the underlying media stream comprises audio,and wherein the transition frames comprise silent frames.
 17. The mediaclient of claim 11, wherein the media client is a television.
 18. Themedia client of claim 11, wherein the media client is a set top box. 19.Non-transitory data storage storing program instructions executable by aprocessing unit to cause a media client to carry out operationsincluding: while the media client is playing out an underlying mediastream, determining a target time at which the media client is to beginplayout of replacement content in place of a portion of the underlyingmedia stream; detecting that a starting time at which the media clientstarts playout of the replacement content is or will be after thedetermined target time by a delay period of duration P; and responsiveto the detecting, reducing playout of the replacement content to helpalign an end of playout of the replacement content with an end of theportion of the underlying media stream, wherein reducing playout of thereplacement content includes (i) detecting in the replacement contentone or more duplicate frames, (ii) responsive to detecting in thereplacement content the one or more duplicate frames, skipping playoutof the one or more duplicate frames when playing out the replacementcontent, (iii) detecting by the device that skipping playout of the oneor more duplicate frames shortens the playout of the replacement contentby less than the duration P, and (iv) responsive to the detecting thatthe removal of the duplicate frames shortens the playout of thereplacement content by less than the delay period of duration P,truncating by the device playout of the replacement content such thatthe playout of the replacement content lasts a duration D-P to helpalign the end of playout of the replacement content with the end of theportion of the underlying media stream.
 20. The non-transitory datastorage of claim 19, wherein the portion of the underlying media streamis an ad in an ad pod, and wherein the replacement content is areplacement ad.